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5^ . Abstract 

The maximum entropy approach operating with quite general entropy mea- 
a • sure and constraint is considered. It is demonstrated that for a conditional or 

c/3 . parametrized probability distribution /(x|/u) there is a "universal" relation among 

the entropy rate and the functions appearing in the constraint. It is shown that the 
recently proposed variational formulation of the entropic functional can be obtained 
as a consequence of this relation, that is from the maximum entropy principle. This 
resolves certain puzzling points appeared in the variational approach. 



1 Introduction 

The maxiinuin entropy approach [T] is extensively invoked in the solution of inference 

OsJ ! problems appearing in various fields (from physics to biology). The approach allows one 

to calculate a probability distribution of relevant variables from an incomplete knowledge 

of the microscopic mechanisms governing the system evolution. The central idea is to find 

Q ■ a least biased distribution consistent with the information at hand. Therefore, the scheme 

O . operates with two main ingredients: the entropy measure (that estimates the uncertainty 

associated with the probability distribution) and the constraints encoding the available 

information. In this context various entropy functionals are introduced (see p] for a recent 

/\ ' review) in order to derive various non-exponential probability distributions observed in 

c^ ■ nature. These are generalizations of the famous Shannon form fITB]) that appears from the 

generalized forms as some limiting case. However, many of these generalized entropies are 

non-additive for statistically independent subsystems. This fact with its consequences is 

still a subject of extensive debates [2]. 

In a situation when many generalized entropy forms are being proposed [3] it might 
seem that the maximum entropy approach suffers from an ambiguity |4]: "in a sense that 
any probability distribution seems to be derivable from the maximization of any entropic 
measure if an appropriate constraint is used" . In this respect it has been argued [^ that 
such an ambiguity does not appear when one searches for a parametric family of distri- 
butions f{x\fi). Here x is a relevant fluctuating variable and n is an external parameter 
independent of x. For instance, such a parameter could enter through the mean value [5] 
or the distribution width varying in response to some external perturbation, like inser- 
tion into a host matrix |6j. On the other hand, we have to note that we are not free to 



modify the constraints without logical necessity because they contain the information (or 
our theoretical model) on the processes running in the system of interest. For example, a 
g— exponential distribution can be easily obtained through maximizing the Tsallis entropy 
flT7|) with the constraint flT51) on the mean value of the relevant variable, but the physics 
is hidden in the form of the entropy functional and the meaning of the index q remains 
unclear. The same g— exponential distribution has been obtained within the superstatis- 
tical approach [7] that operates with two subsystems fluctuating at different time scales. 
On the other hand, it has been demonstrated [HI E] that the same distribution can be ob- 
tained from the Shannon form fll6p with quite transparent physical constraints encoding a 
departure from the standard equilibrium conditions. Namely, a reservoir with fluctuating 
temperature in contact with a system whose phase space is restricted. This corresponds 
to a non-equilibrium stationary state and the magnitude of q measures the distance from 
the equilibrium corresponding to the standard exponential (Gibbsian) distribution. A 
quite similar idea of non-equilibrium hyperensembles, leading to non-exponential distri- 
butions, has recently been proposed y^. All these arguments demonstrate that a given 
distribution can appear as a result of various (not necessarily similar) processes. 

In this connection an inverse problem has been proposed [HI [121 [13]. This variational 
scheme allows one to find the entropy functional that should be chosen in order to find 
a given distribution with a given constraint. It has been found surprising [12] that for 
various distributions the functionals derived in this approach coincide with the generalized 
entropy forms used in the maximum entropy procedure. 

In this paper we consider the maximum entropy procedure operating with quite ar- 
bitrary entropy functional and the constraint imposed on the relevant variable x. This 
allows us to derive a relation among the entropy rate, the observed function and the 
average of its model estimation. This result does not depend (explicitly) neither on the 
entropy form nor on the form of the constraint. Based on this relation we make a link 
between the maximum entropy approach and the variational scheme [TTl [T2| [T3] . This 
explains why the results of the direct (maximum entropy) and the inverse (variational) 
problems are coherent. 

2 Maximum entropy approach 

In the light of what is discussed above, let us consider the entropy functional H{ij) of a 
quite general form 

H{^^) = D{^[f{x\^l)]) (1) 

Here H is simultaneously a function of the parameter /i and a functional of the parametrized 
distribution /(a:|/i), D{t) is some function with appropriate behavior. The functional 
$ [/(x|/i)] can be represented as 



$[/(x|/i)]=ydxv.(/(x|/x)), (2) 

where '^{p) is another function ensuring suitable overall properties (continuity, concav- 
ity,...). Combining eqs. ([T]) and (|2]) we arrive at 

Hifi) = D(^ldxififix\fi))^ (3) 



It is clear that practically all generalized entropy functionals appearing in the current 
literature can be deduced from the definition above (except the forms involving the dis- 
tribution derivatives like the Fisher form). For instance, for D{t) = t, (p{p) = —plnp we 
recover the Shannon entropy fITBl) . The choice D{t) = t, (p{p) = (p— p'^)/(q'— 1) corresponds 
to the Tsallis form (TT7|1 . The Renyi form is recovered if D{t) = Int, <f{p) = p'^/(l — q)- 

Let us consider a problem of maximizing H{n) under two natural constraints - the 
normalization 

dxf{x\fi) = 1 (4) 



and some known average 6{fi) of a conditional response function 6{fi\x) 



eifi) = e{ij\x) = / dxf{x\i2)e{i2\x). (5) 

Here the overbar denotes the corresponding average taken with the distribution f{x\fi). 
Usually the average 9{fi) corresponds to the observed behavior (e.g. experimental data) 
and the conditional function 6{fi\x) represents our theoretical estimation (a model) of this 
behavior for a given state x of the random environment. The latter is characterized by 
a probability distribution f{x\n) depending on the "driving" parameter n (for instance, 
this could be an external field, pressure or chemical potential). 
Thus we have to find a variation of the following Lagrangian 



L = H{fi) + K 



dxf{x\fi)9{^\x) — 9{fi) 



+ A 



dxf{x\fi) 



(6) 



where k and A are the Lagrange multipliers which should be found from the constraints 
(jlj) and (E]). The variation procedure 6L/6f{x\fi) = leads to 

aD($[/(x|/x)])gy.(/(x|/i)) _ 

Let us consider now the variation of H{fi) with respect to the parameter /x. From eqs. 
([1]) and ([2]) we obtain 

dHifi) _dDi<!>[fix\fi)])d^[fix\fi)]_ a$[/(x|/x)]_ f^Jipifix\fi))dfix\fi) ^^^ 



(9/i (9$[/(x|/i)] dfi dfi J df{x\fi) dfi 

Here dD ($ [/(a;|/i)]) /(9$ [/(x|yu)] could be a function of /i. Anyway it does not depend 
on X since $ [/(x|/i)] is a functional of the probability distribution and simultaneously a 
function of fi. Therefore, combining these two equations we obtain 

9H{^i) _ f ^^ dP ($ [fix\^)]) dip {f{x\^i)) dfjxl^i) 



dfi J 9$[/(x|^)] dfixlfi) dfi 

Using now the condition ([7]) for the maximum entropy we arrive at 

^^(^) - -. f dx9i^.\x)^l^ - X fdx^l^ (10) 



It is clear that the last term disappears because of the normalization condition (jlj) and 
we finally get the maximum entropy rate as 

-9l^ = -Jdxe{f.\x)^^. (11) 



Now differentiating the constraint ([5]) we obtain 



9^(/.) _ f^^doi^^^^^^^ ^ f dxei^^ixfJp^ (12) 



Combining these two equations we arrive at 



djj, 



de{i2) de{fi\x) 



(13) 



Therefore, we have a "universal" relation among the entropy rate, the observed behavior 
and the average of the model estimation. This relation does not depend on the form of the 
entropic functional (the form ([1]) is quite general) and thus could be helpful for drawing 
conclusions of general validity. A brief discussion of ( !T3|) for the case of the Shannon form 
can be found in [SI [H] . A more systematic analysis of this issue is left for future studies. In 
what follows we are focusing on a relation between our results and the recently proposed 
variational definition of the entropy functional [12] . 



3 Relation to the variational entropic form 

For a probability distribution f{x) of a random variable x a variational entropic form 

dl = r] dx — dx =7] dxxdf{x) (14) 



has recently been proposed [12] as a measure of uncertainty. Here the overbar again 
denotes the corresponding averages (we will consider continuous variables) 

X = dxxf{x) (15) 

and 1] is a constant of the definition. This form has been used for solving an inverse 
problem, that is finding a functional form of / from a given distribution f{x). It has been 
found that for an exponential distribution the functional / (found from flT^ ) is of the 
Shannon form 

I = - [ dxf{x)\nf{x) (16) 



and for a g— exponential distribution the functional / is of the Tsallis (earlier known as 
Havrda-Charvat) form 

I = -J— fdx[f{x)-r{x)] (17) 

1 — q J 

Surprisingly, this is coherent with the results of the direct problem. Namely, maximizing 
the Shannon and Tsallis functionals with the constraint (llSp gives, respectively, an ex- 
ponential and g— exponential distributions. The author claims [12] that "this is not an 
ordinary and fortuitous mutual invertibility, since the probability and the entropy are not 
reciprocal functions" . 

In what follows we are going to demonstrate that there is a deep link between the 
maximum entropy approach and the variational definition flT^ . It should be noted that 
before starting the discussion it is necessary to specify the meaning of the distribution 
variation df{x) in JHM . In the spirit of the inverse problem mentioned above the variation 



means a change of the functional form of f{x). In the context of our study it is a variation 
of the conditional distribution shape f{x\fi) with the parameter fi. Starting from eq. flTTj) 
and taking into account that dA{fi) = [dA{fi)/dfi]dfi we recover the variational form quite 
similar to flT^ 

dH{fi) = -fi f dxe{fx\x)df{x\fi) (18) 

In contrast to (fT^ . here k is not an arbitrary constant as rj in eq. (fT^ . It is the La- 
grange multiplier associated to the constraint ([5]). Note that in deriving this equation 
the maximum entropy condition (jTj) has been used. Therefore we have a relation between 
the entropy and the distribution which it maximizes. In other words, we have demon- 
strated that the definition fll4p is consistent with the maximum entropy principle for a 
quite arbitrary entropy form. Thus, starting from flT^ one can try to recover the entropy 
functional that is maximized by a given distribution with a given constraint (if such an in- 
verse problem has some practical interest [HI [12]). In this respect it is not surprising that 
the functionals found fTIj [13] from f[T^ with a set of probability distributions coincide 
with the functionals giving these distribution in the maximum entropy approach. Namely, 
considering (TT8|) as a definition of the so-called maximizable entropy H{fi) = H[f{x\^)] 
and following the scheme proposed in fi2[ [13] , for the formally exponential distribution 

/(3;|^) = exp(-^g(/x|x)) 

/ dx exp(— /t^(/i|x)) 

we obtain the Shannon form for the entropy 

H[f{x\n)] = - f dxf{x\fx) In /(x|/i) + const (20) 

Nevertheless, if the relevant variable is x, then the actual form of the distribution (I19p 
depends on the form of the constrained function 6{^\x). For instance, as we have demon- 
strated pi [9], if 0{n\x) is of logarithmic form, then f[T9|) transforms into a F-distribution. 
Physically this corresponds to a non-equilibrium stationary state whose thermodynamic 
entropy is held at a given distance from the maximum that corresponds to the equilib- 
rium. For 6{n\x) = g{fi)x'^ we obtain from f lT^ a stretched exponential distribution. This 
means that we can associate exponential distributions to the Shannon entropy only in the 
case of linear constraint ( ITSjl . i.e. 9{^\x) = g{^)x. 

4 Conclusion 

The maximum entropy procedure operating with a quite general entropic functional is 
considered. This allows us to make a link between the maximum entropy approach and 
the variational formulation [121 US] of the inverse problem. It is shown that in general 
one cannot uniquely associate a given distribution to a given entropic functional because 
a form of the constrained function {9{fj,\x) in our case) also contributes to the actual form 
of the distribution. On the other hand, we are not free to change the constraint in ad hoc 
manner without some physical (or logical) reasons because it encodes our partial infor- 
mation 9{ii) into what we are supposed to know {9{fi\x)) and what we are searching for 
(/(x|/i)). On the author's opinion a more constructive way is to find a well axiomatically 
established ("universal") entropy functional suitable for all applications. Then the strat- 
egy would be quite clear. That is, if a given distribution cannot be obtained through the 



maximization of the entropy with a given constraint, then the information (or physics) en- 
coded in the constraint is not compatible with the observed behavior. In such a situation 
one has to search for another scenario (represented hj 6{fi\x)) of the processes running in 
the system of interest. 
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